Systems and methods for training statistical speech translation systems from speech

ABSTRACT

An iterative language translation system. The system includes a first automatic speech recognition component adapted to recognize spoken language in a source language and to create a source language hypothesis and a first machine translation component adapted to translate the source language hypothesis into a target language. The system also includes a second automatic speech recognition component adapted to recognize spoken language in the target language that is spoken by a translator, and wherein the second automatic speech recognition component is further adapted to create a target language hypothesis. The system further includes a second machine translation component adapted to translate the target language hypothesis into the source language, wherein the translation of the target language hypothesis into the source language is used to adapt the first automatic speech recognition component, wherein the translation of the source language hypothesis into the target language is used to adapt the second automatic speech recognition component, wherein the source language hypothesis is used to adapt the first machine translation component and the second machine translation component, and wherein the target language hypothesis is used to adapt the first machine translation component and the second machine translation component.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to U.S. Provisional PatentApplication No. 60/802,873 filed May 22, 2006.

BACKGROUND

Speech translation systems combine recognition of speech withtranslation from one language (“source language”) to another language(“target language”) followed by optional synthesis or text output in atarget language. The development of such systems requires development ofhigh performance speech recognition systems and translation systems. Fortheir development, these systems require substantial data resourcesbased on how the recognition and translation engines are trained ordeveloped. Thousands of spoken sentences have to be transcribed, andthousands of sentences in one language have to be translated intoanother. Moreover, data collection has to be redone with each newlanguage and, when necessary, for different domains and genres.

Thus, there is a need for methods and apparatuses that allow speechtranslation systems to be trained or to “learn” from examples providedby human simultaneous translators. There is a further need for methodsand apparatuses in which speech data in both source and target languagesare presented arid which speech and translation engines iterativelylearn together, thereby foregoing the labor intensive and costly stepsof annotating data from speech and translating texts and of training andoptimizing the speech recognition and translation engines independentlyfirst before system combination is attempted. Also, there is a need fora field-correctable translation system in which a person in the field ofuse can correct errors made by the system so that the system will adapt.There is a further need for a translation system that is adept attranslating languages for which there is not a large written corpus.

SUMMARY

In various embodiments, the present invention is directed to aniterative language translation system. The system includes a firstautomatic speech recognition component adapted to recognize spokenlanguage in a source language and to create a source language hypothesisand a first machine translation component adapted to translate thesource language hypothesis into a target language. The system alsoincludes a second automatic speech recognition component adapted torecognize spoken language in the target language that is spoken by atranslator, and wherein the second automatic speech recognitioncomponent is further adapted to create a target language hypothesis. Thesystem further includes a second machine translation component adaptedto translate the target language hypothesis into the source language,wherein the translation of the target language hypothesis into thesource language is used to adapt the first automatic speech recognitioncomponent, wherein the translation of the source language hypothesisinto the target language is used to adapt the second automatic speechrecognition component, wherein the source language hypothesis is used toadapt the first machine translation component and the second machinetranslation component, and wherein the target language hypothesis isused to adapt the first machine translation component and the secondmachine translation component.

In various embodiments, the present invention is directed to aniterative method of translating language from a source language to atarget language. The method includes recognizing, with a first automaticspeech recognition component, spoken language in a source language andto create a source language hypothesis and translating, with a firstmachine translation component, the source language hypothesis into atarget language. The method also includes recognizing, with a secondautomatic speech recognition component, spoken language in the targetlanguage that is spoken by a translator and creating, with the secondautomatic speech recognition component, a target language hypothesis.The method further includes translating, with a second machinetranslation component, the target language hypothesis into the sourcelanguage, adapting the first automatic speech recognition componentusing the translation of the target language hypothesis into the sourcelanguage, and adapting the second automatic speech recognition componentusing the translation of the source language hypothesis into the targetlanguage. The method also includes adapting the first machinetranslation component and the second machine translation component usingthe source language hypothesis and adapting the first machinetranslation component and the second machine translation component usingthe target language hypothesis.

In various embodiments, the present invention is directed to aniterative language translation system. The system includes an automaticspeech recognition component adapted to recognize spoken language in asource language and to create a source language hypothesis and a machinetranslation component adapted to translate the source languagehypothesis into a target language. The system also includes a universalspeech recognition component adapted to recognize spoken language in anylanguage, and wherein the universal speech recognition component isfurther adapted to create a representation of target language speech,wherein the translation of the target language hypothesis into thesource language is used to train the automatic speech recognitioncomponent, wherein the translation of the source language hypothesisinto the target language is used to train the universal speechrecognition component, wherein the source language hypothesis is used totrain the machine translation component, and wherein the representationof target language speech is used to train the machine translationcomponent.

In various embodiments, the present invention is directed to aniterative language translation system. The system includes an automaticspeech recognition component adapted to recognize spoken language in asource language and to create a source language hypothesis and a machinetranslation component adapted to translate the source languagehypothesis into a target language. The system also includes a universalspeech recognition component adapted to recognize spoken language in anylanguage, and wherein the universal speech recognition component isfurther adapted to enhance a target language hypothesis that has beencreated from a monolingual extract of text material in the targetlanguage, wherein the translation of the target language hypothesis intothe source language is used to train the automatic speech recognitioncomponent, wherein the translation of the source language hypothesisinto the target language is used to train the universal speechrecognition component, wherein the source language hypothesis is used totrain the machine translation component, and wherein the target languagehypothesis is used to train the machine translation component.

In various embodiments, the present invention is directed to aniterative language translation system for translating a source languageto a target language. The system includes first recognition means forrecognizing spoken language in a source language and for creating asource language hypothesis, first translation means for translating thesource language hypothesis into a target language, and secondrecognition means for recognizing spoken language in the target languagethat is spoken by a translator. The system also includes means forcreating a target language hypothesis, second translation means fortranslating the target language hypothesis into the source language, andmeans for adapting the first recognition means using the translation ofthe target language hypothesis into the source language. The systemfurther includes means for adapting the second recognition means usingthe translation of the source language hypothesis into the targetlanguage, means for adapting the first translation means and the secondtranslation means using the source language hypothesis, and means foradapting the first translation means and the second translation meansusing the target language hypothesis.

In various embodiments, the present invention is directed to a computerreadable medium having stored thereon instructions which, when executedby a processor, cause the processor to:

recognize, with a first automatic speech recognition component, spokenlanguage in a source language and to create a source languagehypothesis;

translate, with a first machine translation component, the sourcelanguage hypothesis into a target language;

recognize, with a second automatic speech recognition component, spokenlanguage in the target language that is spoken by a translator;

create, with the second automatic speech recognition component, a targetlanguage hypothesis;

translate, with a second machine translation component, the targetlanguage hypothesis into the source language;

adapt the first automatic speech recognition component using thetranslation of the target language hypothesis into the source language;

adapt the second automatic speech recognition component using thetranslation of the source language hypothesis into the target language;

adapt the first machine translation component and the second machinetranslation component using the source language hypothesis; and

adapt the first machine translation component and the second machinetranslation component using the target language hypothesis.

Those and other details, objects, and advantages of the presentinvention will become better understood or apparent from the followingdescription and drawings showing embodiments thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate examples of embodiments of theinvention. In such drawings:

FIG. 1 illustrates an embodiment of an iterative speech translationsystem;

FIG. 2 illustrates an overview of the performance characteristics of anEnglish and Spanish baseline ASR system;

FIG. 3 illustrates an embodiment of a document driven system;

FIG. 4 illustrates an overview of the components of a document driveniterative system along with respective performance values according toone embodiment of the present invention;

FIG. 5 illustrates an overview of the components of a speech driveniterative system along with respective performance values according toone embodiment of the present invention;

FIG. 6 illustrates a detailed comparison of the performance of anEnglish ASR system in the document driven case and the speech drivencase;

FIG. 7 illustrates the best and worst performing speakers within the twoEnglish ASR subsystems before applying MT knowledge and after applyingMT knowledge with the help of an iterative scheme;

FIG. 8 illustrates an overview of the performance of baseline ASRsystems; and

FIG. 9 illustrates an embodiment of a system in which multiple audiostreams of human simultaneous translators are used.

DESCRIPTION

Various embodiments of the present invention describe methods andapparatuses that permit automatic training of speech translation systemsfrom field input and from examples gathered from human simultaneoustranslators. In various embodiments, translation systems may be trainedbased on a parallel corpus of spoken speech in, for example, twolanguages (e.g., recorded speech from a speaker and from an interpreterwho interprets the speech). Various embodiments reduce development time,because it is not necessary to collect and transcribe speech data andtranslation data. Various embodiments of a method for adapting andtraining statistical speech translation systems directly and in anunsupervised manner from the speech of human translators, including inparticular the training of machine translation systems from speech aredescribed.

Various embodiments of the present invention are directed to anunsupervised training scheme for statistical speech translation. As usedherein, the term “speech translation system” refers to any system orsoftware that provides translation of speech in one source language totext in another target language. As used herein, the term “trainingdata” includes, for example, monolingual text data for language modeltraining, sentence aligned bilingual text data for translation modeltraining, or audio data along with the transcripts for acoustic modeltraining. Automatic speech translation consists of two steps: automaticspeech recognition for transcribing the speech in the source languageand automatic translation for translating the transcribed sourcelanguage speech into the target language. According to these two steps,speech translation systems traditionally consist of two separate systemcomponents—the speech recognition system and the machine translationsystem. Automatic speech recognition (ASR) systems and machinetranslation (MT) systems rely on learning (e.g., statistical learning)models for recognizing speech or translating text. The statisticalmodels include the acoustic model for speech recognition and thetranslation model for machine translation. In addition, both systems(e.g., in the case of large vocabularies) rely on statistical n-gramlanguage models. To estimate good statistical models, large corpora oftraining data are needed. These corpora consist of transcribed speechdata (e.g., audio files of speech together with their transcriptions intext form) for estimating the acoustic model and of aligned bilingualtext data for estimating the translation model.

As used herein, the term “supervised training” means that thecorresponding statistical models are trained under human supervision asthe training corpora are created by human transcribers or humantranslators. For the estimation of acoustic models unsupervised trainingmethods are also often used in which an already existing speechrecognition system is used to automatically transcribe speech data. Thetranscribed speech data are then used to further refine the acousticmodel. Similarly, adaptation of speech recognizers have beenaccomplished and improved using automatically translated bilingual data.

Various embodiments of the present invention are directed to a trainingand adaptation scheme that permits direct training and/or adaptation ofcomplete statistical speech translation systems from speech in anunsupervised manner. As used herein, the term “complete” refers to thesituation in which all involved statistical models are jointly refined.This includes, for example, an unsupervised training of statisticaltranslation models from the speech of human translators. Variousembodiments of the methods described herein are iterative and rely onthe availability of the target language speech of one or more humantranslators and on the availability of the source language speech or asource language text document that is being translated by the humantranslator(s). An example training scenario used by embodiments of thepresent invention would be the simultaneous translation of a speechwithin the United Nations or the European Parliament, where the speechof the simultaneous translator and the speech of the lecturer can berecorded.

In the training scheme, two different training situations can bedistinguished in various embodiments—one of them document driven (seeFIG. 3) and the other one speech driven (see FIG. 1). In the documentdriven case a human translator speaks the translation of a textdocument. In the speech driven case a human translator speaks thetranslation of source language speech during, for example, asimultaneous translation.

As illustrated in FIG. 1, according to various embodiments the trainingscheme relies in the speech driven case on two (or more) statisticalspeech translation systems, each consisting of an automatic speechrecognition (ASR) component 10, 12 and a machine translation (MT)component 14, 16. Both speech translation systems are connected in aniterative, overall system 18 to allow a recursive adaptation andtraining of all involved system components. In the system 18, allavailable knowledge sources, namely the speech in the source and targetlanguage, can be incorporated and used for adaptation and training ofthe different system components (i.e., for adaptation and training oftheir attached statistical models).

The adaptation and training of the system components of the system 18works as follows in various embodiments: starting with source languagespeech recognition hypotheses 20 and their automatically created targetlanguage translations, it is possible to bias the target language speechrecognition system 12 towards the gained knowledge, e.g. by adapting theASR language model. Because the target language machine translationhypotheses 22 can be seen as a prediction of what a human translator 24will say, the recognition accuracy of the biased target ASR system 12will improve. As a result, the target language ASR hypotheses 22 will beof a higher quality (less erroneous).

The high quality target language ASR hypotheses 22 can then be used intwo different ways: as additional training data for the target languageASR system 12 (for a further training or adaptation of the acousticmodels and the ASR language model) and as additional training data forthe source to target MT 16 and the target to source MT 14. The usage asadditional training data for both machine translation systems 14, 16 ispossible because the translation hypotheses 22, 24 together form analigned bilingual corpus, suitable for training the respectivetranslation models. Also, the hypotheses 22, 24 can be used asadditional training or adaptation data for the respective targetlanguage models used within the two machine translation systems 14, 16.

In various embodiments, it is possible to improve the performance of thetarget ASR system 12 with the automatically translated source hypotheses20 and it is again possible to improve the performance of the source ASRsystem 10 by automatically translating the target ASR hypotheses 22.Thus, the system 18 performs in an iterative maimer and the iterativecycle can be traversed x times on the same data. In various embodiments,the value of x can be estimated in two ways, depending on the ultimategoal for applying the iterative scheme. First, the iterative cycle canbe traversed until a saturation in the improvement/a decrease inperformance of the different statistical models on a control set isobserved. As used herein, the term “control set” refers to a small dataset with known transcripts on which the same iterative training schemeis applied. The stopping criterion is applied in order to maximallyadapt the involved systems towards the given data so that a maximalquality for the ASR hypotheses 20, 22 can be accomplished. The secondpossible stopping criterion is to traverse the iterative cycle until theperformance on a held out data set starts to decrease, therebyindicating an overfitting of the statistical models towards the usedtraining data. As used herein, the term “held out data” refers to a dataset of the same or similar domain as the training data, whereas theiterative training scheme is not being applied to this data. The secondstopping criterion is applied to train the respective statistical models(ASR models and MT models) so that the training data is utilized in anoptimal way.

In various embodiments, the iterative nature of the system 18 allows forthe incorporation of knowledge provided by not just one audio stream inanother language, but by many. An example is the simultaneoustranslation of a European Parliament speech into multiple languages by amultitude of human translators. Also, while embodiments of the presentinvention automatically provide an improvement (in the sense of anadaptation or an additional training) of all involved systemscomponents, in various embodiments the adaptation/training may besteered towards a more optimal utilization of the available data for one(or a specific subset) of the involved system components.

Various embodiments of the present invention may be used for trainingand/or adaptation of speech translation systems or “focused” trainingand/or adaptation of one of the system components of a speechtranslation system (either ASR or MT). Various embodiments may be usedfor adaptation and training of speech recognition systems for humantranslators and ASR systems as a tool for human translators to speed uptranslations with the intended use of the translations in text form forpublication or archiving. Various embodiments may be used for rapidadaptation of existing speech translation systems to new domains andrapid development of speech translation systems for resource deficientlanguages (“learning by doing”). Also, various embodiments may be usedfor correcting mistakes in existing speech translation systems, andeffective adapting and improving system performance during use.

An implementation of the system 18 was constructed with Spanish as thesource language and English as the target language. The data set used+consisted of 500 parallel English and Spanish sentences in form andcontent close to the Basic Travel Expression Corpus (BTEC) described inG. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto, “Creating Corpora forSpeech-to-Speech Translation,” Proceedings of Eurospeech, Geneve,Switzerland, 2003. The sentences were presented two times, each timeread by three different Spanish and five different English speakers. Tenpercent of the data was randomly selected as heldout data for systemparameter tuning. Parameter tuning was done by mammal gradient descentthroughout this work. Because of some flawed recordings, the Englishdata set had 880 sentences with 6,751 (946 different) words. Therespective Spanish data set had 900 sentences composed of 6,395 (1,089different) words. The Spanish audio data equaled 45 minutes and theEnglish 33 minutes.

Because the sentences were presented two times there were always two ASRhypotheses for each sentence, decoded on the speech of two differentspeakers. Using both of these hypotheses within the iterative system 18would change the system 18 into a voting system that chooses between thetwo hypotheses. Thus, the data set was split into two disjoint parts, sothat each Spanish-English sentence pair occurred only once within eachsubset. Based on the two subsets, two different iterative systems had tobe examined. In the following discussion, only the average performance,calculated on the two individual system results, is given.

For the example embodiment of the system 18, a Janus Recognition Toolkit(JRTk) featuring the IBIS single pass decoder, as described in H.Soltau, F. Metze, C. Fugen, and A. Waibel, “A One-Pass Decoder Based onPolymorphic Linguistic Context Assignment,” Proceedings of ASRIJ,Madonna di Campiglio, Italy, 2001, was used. FIG. 2 illustrates anoverview of the performance characteristics of the English and Spanishbaseline ASR system.

In the example implementation, the English speech recognition system 12was a subphonetically tied semi-continuous three-state HMM based systemthat had 6K codebooks, 24K distributions and a 42-dimensional featurespace on MFCCs after LDA. It used semi-tied covariance matrices,utterance-based CMS and incremental VTLN with feature-space constrainedMLLR. The vocabulary size was 18K. The recognizer was trained on 180 hBroadcast News data and 96 h Meeting data. The back off tri-gramlanguage model was trained on the English BTEC which consisted of 162.2Ksentences with 963.5K running words from 13.7K distinct words.

Spanish recognizer 10 had 2K codebooks and 8K distributions; all othermain characteristics were equivalent to the characteristics of theEnglish recognizer. The vocabulary size was 17K. The system was trainedon 112 h South American speech data (mainly Mexican and Costa Ricandialects) and 14 h Castilian speech data. The South American corpus wascomposed of 70 h Broadcast News data, 30 h Globalphone data and 12 hSpanish Spontaneous Scheduling Task data. The back-off tri-gram LM wastrained on the Spanish part of the BTEC.

In the example implementation, the ISL statistical machine translationsystem described in S. Vogel, S. Hewavitharana, M. Kolss, and A. Waibel,“The ISL Statistical Machine Translation System for Spoken LanguageTranslation,” Proceedings of IWSLT, Kyoto, Japan, 2004, was used forcreating the English-to-Spanish 14 and Spanish-to-English translations16. The MT system was based on phrase-to-phrase translations (calculatedon word-to-word translation probabilities) extracted from a bilingualcorpus, i.e., the Spanish/English BTEC. The MT system produces an n-bestlist of translation hypotheses for a given source sentence with the helpof its translation model (TM), target language model and translationmemory. The translation memory works as follows: for each sourcesentence that has to be translated the closest matching source sentence,with regard to the edit distance, is searched in the training corpus andextracted along with its translation. In case of an exact match theextracted translation was used, otherwise different repair strategieswere applied to find the correct translation. The translation modelcomputed the phrase translation probability based on word translationprobabilities found in its statistical IBM1 forward and backward lexicaregardless of the word order. The word order of the MT hypotheses wastherefore appointed by the LM and translation memory. Because the MT andthe ASR used the same language models, only the translation memory canprovide additional word order information for improving the ASR.

Various embodiments of the present invention employ ASR improvementtechniques. In one embodiment, for hypothesis selection the 150 best ASRhypotheses of the ASR system are used together with the first best MThypothesis of the MT system preceding this ASR system within theiterative cycle. The applied rescoring algorithm computes new scores(negative log-probabilities) for each of the 151 sentences by summingover the weighted and normalized ASR score (s_(ASR)), language modelscore (s_(LM)), and translation model score (s_(TM)) of the sentence. Tocompensate for the different ranges of the values for the TM, LM and ASRscores, the individual scores in the n-best lists are scaled to [0; 1].

s _(final) =s′ _(ASR) +w _(LM) *s _(LM) +w _(TM) *s _(TM)   (1)

The ASR score output by the JRTk is a linear combination of acousticscore, scaled language model score, word penalty lp and filler wordpenalty fp. The language model score within this linear combinationcontains discounts for special words or word classes. The rescoringalgorithm allows to directly change the word penalty and the filler wordpenalty added to the acoustic score. Moreover, four new word contextclasses with their specific LM discounts may be used: MT mono-, bi-,trigrams and complete MT sentences (the respective LM discounts are md,bd, td and sd). MT n-grams are n-grams included in the respective MTnbest list; MT sentences are defined in the same manner. The ASR scorein equation (1) is therefore computed as:

s′ _(ASR) =s _(ASR) +lp′*n _(words) +fp′*n _(fillerwords)

−md*n_(MTmonograms)−bd*n_(MTbigrams)

−td*n_(MTtrigrams)−sd*δ_(isMTsentence)   (2)

The rescoring approach applies MT knowledge in two different ways: bycomputing the TM score for each individual hypothesis and by introducingnew word class discounts based on MT n-best lists. It has been shownthat the MT mono-gram discounts have the strongest influence on thesuccess of the rescoring approach, followed by the TM score. Otherparameters apart from the mono-gram discount md and translation modelweight w_(TM) only have inferior roles and can be set to zero. Thissuggests that the additional word context information in form of MT bi-and tri-grams may not be very useful for improving the ASR. However, theMT component is very useful as a provider for a “bag-of-words” thatpredicts which words are going to be used by the human translator.

A classical cache language model has a dynamic memory component thatremembers the recent word history of m words to adjust the languagemodel probabilities based on the history. The cache LM used in variousembodiments of the present invention has a dynamically updated “cache,”whereas the LM probabilities are influenced by the content of the cache.However, the cache is not used to remember the recent word history butto hold the words (mono-grams) found in the respective MT n-best list ofthe sentence that is being decoded at the moment. The cache LM isrealized by defining the members of the word class mono-gram in the samemanner as for the rescoring approach, but now dynamically, duringdecoding. Within the basic ASR improvement techniques, the cache LMapproach may yield the best improvements results, closely followed bythe rescoring approach. This result helps validate the usefulness of the“bag-of-words” knowledge provided by the MT. As this “bag-of-words”knowledge is already applied during decoding, new correct hypotheses arefound due to positive pruning effects. This explains why the cache LMapproach is able to slightly outperform the rescoring approach, althoughit lacks the additional form of MT knowledge used by the rescoringapproach, namely the direct computation of the TM score.

For language model interpolation, the original LM of the ASR system isinterpolated with a small back-off tri-grain language model computed onthe translations found within all MT n-best lists. LM interpolationyields only small improvements compared to the cache LM and therescoring approach. This can be explained by the little value of MT wordcontext information for ASR improvement described hereinabove.

Similar to the improvement of the ASR, the MT improvement techniquewithin the iterative system described in connection with variousembodiments herein is a combination of two basic MT improvementtechniques, namely language model interpolation and MT systemretraining. For language model interpolation, the original MT languagemodel is interpolated with a small back-off tri-gram language modelcomputed on the hypotheses found within all ASR n-best lists. MT systemretraining is done by adding the ASR n-best lists several times to theoriginal training data and computing new IBM1 lexica (forward andbackward lexicon), whereas the translation memory component of the MTsystem is held fixed to the original training data. The reason forkeeping the translation memory fixed is that an updated memory leads toa loss of complementary MT knowledge that is valuable for further ASRimprovement. An updated memory sees to it that the ASR n-best hypothesesadded to the original training data are chosen as translation hypothesesby the MT system, meaning that only a slightly changed ASR output of thepreceding iteration is used for ASR improvement in the next iterationinstead of new MT hypotheses.

The LM interpolation contributes the most to the MT improvement if thetranslation memory is kept fixed. This means that, while the wordcontext information provided by the MT is of only minimal use forimproving the ASR, word context information provided by the ASR is veryvaluable to improving the MT.

Different combinations of the basic ASR and MT improvement techniquesdescribed hereinabove were taken into consideration for an embodiment ofa document driven system 26 as shown in FIG. 3. The best results inregard to improving the English ASR system 28 may be observed when usingthe combination of LM interpolation and retraining with a fixedtranslation memory as MT improvement technique. The combination ofrescoring and cache LM in iteration 0 and the combination of rescoring,cache LM and interpolated LM in iteration 1 may yield the best resultsas ASR improvement techniques. The better performance resulting from theadditional use of LM interpolation after iteration 0 is due to theimproved MT context information. The success of the subsequent rescoringof the ASR output is due to the additional form of MT knowledge appliedby the rescoring approach; in contrast to the cache LM approach,rescoring does not only consider the MT “bag-of-words” knowledge butalso considers the TM score. It could be observed that the mostimportant parameter for rescoring on cache LM system output is thetranslation model weight w_(TM), because after setting all otherparameter to zero, still similar good results could be achieved. Nosignificant improvements were observed for iterations >1. This was truefor all examined system combinations that applied a subsequent rescoringon the ASR system output. If no rescoring was used, similar results tothe case where rescoring was used could be obtained, but only afterseveral (>3) iterations.

FIG. 4 gives an overview on the components of a document driveniterative system 40 along with the respective performance values. Withthe iterative approach the WER. of the English baseline ASR system 42may be reduced from 20.4% to 13.1%. This is equivalent to a relativereduction of 35.8%.

Different combinations of the basic ASR and MT improvement techniqueswere taken into consideration for various embodiments of the speechdriven system described herein. As in the document driven case, the MTcomponents were improved just once within the iterative system designfor gaining best results in speech recognition accuracy (for bothinvolved ASR systems). This means that in order to avoid overfitting,the iterative process should be aborted right before an involved MTcomponent would be improved a second time. FIG. 5 gives an overview ofthe components of a speech driven iterative system 50 along with therespective performance values according to one embodiment of the presentinvention. The WER of the English baseline ASR system 52 was reducedfrom 20.4% to 14.3%. This is a relative reduction of 29.9%.

In iteration 0, the BLEU score of the Spanish-to-English MT system 54 is15.1% relative worse than in the document driven case. This is due tothe fact that the Spanish source sentences used for translation nowcontain speech recognition errors. In this context it should be notedthat this loss in MT performance is of approximately the same magnitudeas the WER of the Spanish input used for translation, i.e. it is ofapproximately the same magnitude as the WER of the Spanish baselinesystem. The loss in MT performance leads to a smaller improvement of theEnglish ASR system 52 compared to the document driven case. However, theloss in MT performance does not lead to a loss in English speechrecognition accuracy of the same magnitude; compared to the documentdriven case the WER of the English ASR system 52 is only 9.8% relativehigher. FIG. 6 shows a detailed comparison of the performance of theEnglish ASR system in the document driven case and the speech drivencase. Even though the gain in recognition accuracy is already remarkablyhigh in both cases without applying any iteration, a still significantgain in performance was observed in the first iteration.

As mentioned hereinabove, two different STE-ASR systems may be used, onefor each of the two data subsets. FIG. 7 shows the best and worstperforming speakers within the two English ASR subsystems beforeapplying MT knowledge and after applying MT knowledge with the help ofthe iterative scheme described herein. While the WER of the worstspeaker is reduced by 36.7% relative, the WER of the best speaker isonly reduced by 31.3% relative. This means that for speakers with higherword error rates a higher gain in recognition accuracy is accomplishedby applying MT knowledge.

The ASR driven system described herein automatically provides animprovement of the involved source language ASR. In one implementation,the WER of the Spanish baseline ASR of 17.2% is reduced by 20.9%relative. This smaller improvement in recognition accuracy compared tothe improvement of the English ASR may be explained by the fact thatSpanish is a morphologically more complicated language than English.

It is directly possible to incorporate not just one, but several targetlanguage audio streams into the iterative system of embodiments of thepresent invention. For this, the applied improvement techniques onlyneed to be adapted minimally. The adaption of the cache LM approach aswell as the LM interpolation (for ASR and MT improvement) and MTretraining is done by including all MT/ASR n-best lists of the precedingMT/ASR systems in the iterative cycle. For rescoring, Equation (1) isextended to allow for several TM scores provided by several MT systemswith different target languages, i.e. instead of one TM score andassociated TM weight there are now up to n TM scores with theirrespective TM weights. In the following example, it is shown how analready speech translation enhanced English ASR system is furtherimproved by adding knowledge provided by one additional audio stream ina different target language.

For the implementation a BTEC held-out data set consisting of 506parallel Spanish, English and Mandarin Chinese sentences was used. Tenpercent of the data was randomly selected for system parameter tuning.The English and Spanish sentences were read twice, the Chinese Sentenceswere read just once. The same Spanish and English baseline ASR systemswere used as before. For Chinese speech recognition the ISL RT04Mandarin Broadcast News evaluation system was used. The vocabulary ofthe Chinese ASR system has 17K words. The Chinese LM was computed on theChinese BTEC. FIG. 8 gives an overview of the performance of thebaseline ASR systems.

In one implementation, the Spanish and English audio streams for speechtranslation based ASR improvement were used. The same iterative STE-ASRtechnique was applied as that described hereinabove except no LMinterpolation was used for improving the English ASR system, as aslightly worse WER was observed for doing so. The negative influence ofLM interpolation on the performance of the English ASR system can beexplained by the already very good match of the English baseline LM withthe used data set (the perplexity is only 21.9). The WER of the SpanishASR system was reduced from 15.1% to 13.4%. The WER of the English ASRsystem was reduced from 13.5% to 10.6%. Next, it was determined if theperfonnance of the improved English ASR system could be furtherincreased by taking advantage of the additional Chinese audio streamsFor this, the Chinese baseline system was improved with the help of thelatest computed English system output and then the output of theimproved Chinese system was used to once again improve the Englishsystem. The MT systems for translating between English and Chinese weretrained on the Chinese-English BTEC. The accomplished BLEU scores werewith 21.2 for E→C and with 24.1 for C→E very moderate. Nevertheless, theWER of the Chinese system was reduced from 20.0% to 17.1% and for theEnglish system from 10.6% to 10.3%.

FIG. 9 illustrates an embodiment of a system 60 in which multiple audiostreams of human simultaneous translators are used. In FIG. 9, there aren target languages.

In various embodiments of the present invention, a translation system isimplemented in which there is no corpus of translated material in thetarget language. In such a case, a language universal acousticrecognizer that is adaptive and language independent may be used forspeech recognition. An example of such a recognizer is described in T.Schultz, A. Waibel, “Language Independent and Language Adaptive AcousticModeling for Speech Recognition”, Speech Communication, Vol. 35,February 2001, which is incorporated herein by reference. In such asystem, a source language ASR may be used for the source language inconjunction with the universal language independent recognizer. Both thesource language ASR and the universal recognizer may be adaptive and maybe trained with each iteration of the system. Also, in variousembodiments, when there is no corpus of material in the target language,text extracts in the target language may be used to train a universalrecognizer for the target language. As in other embodiments, theuniversal recognizer may be adaptive and trainable, but its initialaccuracy may be better with such extracts than without. An example of anextract is a web page in the target language that has been translatedinto the source language. The embodiments described herein that use auniversal recognizer may be applicable in situations when the targetlanguage is a language that is rare and, thus, there is a lack oftranslated material in that language.

It can be understood that although many embodiments and implementationsof the present invention have been described as being applicable tocertain languages (i.e., English, Spanish and Chinese), the systems andmethods described herein are applicable to any language. Also,embodiments of the present invention may have particular applicabilityto languages that do not have large corpi of written material.

Various embodiments of the present invention may be implemented oncomputer-readable media. The terms “computer-readable medium” and“computer-readable media” in the plural as used herein may include, forexample, magnetic and optical memory devices such as diskettes, compactdiscs of both read-only and writeable varieties, optical disk drives,hard disk drives, etc. A computer-readable medium may also includememory storage that can be physical, virtual, permanent, temporary,semi-permanent and/or semi-temporary. A computer-readable medium mayfurther include one or more data signals transmitted on one or morecarrier waves.

While the foregoing has been set forth in considerable detail, it is tobe understood that the drawings and detailed embodiments are presentedfor elucidation and not limitation. Design variations may be made butare within the principles of the invention. Those skilled in the artwill realize that such changes or modifications of the invention orcombinations of elements, variations, equivalents, or improvementstherein are still within the scope of the invention as defined in theappended claims.

1. An iterative language translation system, comprising: a firstautomatic speech recognition component adapted to recognize spokenlanguage in a source language and to create a source languagehypothesis; a first machine translation component adapted to translatethe source language hypothesis into a target language; a secondautomatic speech recognition component adapted to recognize spokenlanguage in the target language that is spoken by a translator, andwherein the second automatic speech recognition component is furtheradapted to create a target language hypothesis; a second machinetranslation component adapted to translate the target languagehypothesis into the source language; and wherein the translation of thetarget language hypothesis into the source language is used to adapt thefirst automatic speech recognition component, wherein the translation ofthe source language hypothesis into the target language is used to adaptthe second automatic speech recognition component, wherein the sourcelanguage hypothesis is used to adapt the first machine translationcomponent and the second machine translation component, and wherein thetarget language hypothesis is used to adapt the first machinetranslation component and the second machine translation component. 2.The system of claim 1, wherein at least one of the first automaticspeech recognition component and the second automatic speech recognitioncomponent utilize an applied rescoring technique.
 3. The system of claim1, wherein at least one of the first automatic speech recognitioncomponent and the second automatic speech recognition include adynamically updated cache memory.
 4. The system of claim 1, wherein atleast one of the first automatic speech recognition component and thesecond automatic speech recognition component utilize a language modelinterpolation technique.
 5. The system of claim 1, wherein at least oneof the first machine translation component and the second machinetranslation component utilize a language model interpolation technique.6. The system of claim 1, wherein at least one of the first machinetranslation component and the second machine translation componentutilize a system retraining technique.
 7. The system of claim 1, whereinthe system is adapted to be corrected by a spoken correction during useof the system.
 8. The system of claim 1, further comprising: a thirdautomatic speech recognition component adapted to recognize spokenlanguage in a second target language that is spoken by a secondtranslator, and wherein the third automatic speech recognition componentis further adapted to create a second target language hypothesis; athird machine translation component adapted to translate the secondtarget language hypothesis into the source language; and wherein thetranslation of the second target language hypothesis into the sourcelanguage is used to train the first automatic speech recognitioncomponent, wherein the translation of the source language hypothesisinto the target language is used to train the third automatic speechrecognition component, wherein the source language hypothesis is used totrain the third machine translation component, and wherein the secondtarget language hypothesis is used to train the third machinetranslation component.
 9. The system of claim 1, wherein the sourcelanguage hypothesis is used to train at least one of the first automaticspeech recognition component, the second automatic speech recognitioncomponent, the first machine translation component, and the secondmachine translation component.
 10. The system of claim 1, wherein thetarget language hypothesis is used to train at least one of the firstautomatic speech recognition component, the second automatic speechrecognition component, the first machine translation component, and thesecond machine translation component.
 11. An iterative method oftranslating language from a source language to a target language, themethod comprising: recognizing, with a first automatic speechrecognition component, spoken language in a source language and tocreate a source language hypothesis; translating, with a first machinetranslation component, the source language hypothesis into a targetlanguage; recognizing, with a second automatic speech recognitioncomponent, spoken language in the target language that is spoken by atranslator; creating, with the second automatic speech recognitioncomponent, a target language hypothesis; translating, with a secondmachine translation component, the target language hypothesis into thesource language; adapting the first automatic speech recognitioncomponent using the translation of the target language hypothesis intothe source language; adapting the second automatic speech recognitioncomponent using the translation of the source language hypothesis intothe target language; adapting the first machine translation componentand the second machine translation component using the source languagehypothesis; and adapting the first machine translation component and thesecond machine translation component using the target languagehypothesis.
 12. An iterative language translation system, comprising: anautomatic speech recognition component adapted to recognize spokenlanguage in a source language and to create a source languagehypothesis; a machine translation component adapted to translate thesource language hypothesis into a target language; a universal speechrecognition component adapted to recognize spoken language in anylanguage, and wherein the universal speech recognition component isfurther adapted to create a representation of target language speech;and wherein the translation of the target language hypothesis into thesource language is used to train the automatic speech recognitioncomponent, wherein the translation of the source language hypothesisinto the target language is used to train the universal speechrecognition component, wherein the source language hypothesis is used totrain the machine translation component, and wherein the representationof target language speech is used to train the machine translationcomponent.
 13. The system of claim 12, wherein the representation oftarget language speech includes a phoneme string.
 14. The system ofclaim 12, wherein the universal speech recognition component is alanguage universal acoustic speech recognizer.
 15. An iterative languagetranslation system, comprising: an automatic speech recognitioncomponent adapted to recognize spoken language in a source language andto create a source language hypothesis; a machine translation componentadapted to translate the source language hypothesis into a targetlanguage; a universal speech recognition component adapted to recognizespoken language in any language, and wherein the universal speechrecognition component is further adapted to enhance a target languagehypothesis that has been created from a monolingual extract of textmaterial in the target language; and wherein the translation of thetarget language hypothesis into the source language is used to train theautomatic speech recognition component, wherein the translation of thesource language hypothesis into the target language is used to train theuniversal speech recognition component, wherein the source languagehypothesis is used to train the machine translation component, andwherein the target language hypothesis is used to train the machinetranslation component.
 16. An iterative language translation system fortranslating a source language to a target language, the systemcomprising: first recognition means for recognizing spoken language in asource language and for creating a source language hypothesis; firsttranslation means for translating the source language hypothesis into atarget language; second recognition means for recognizing spokenlanguage in the target language that is spoken by a translator; meansfor creating a target language hypothesis; second translation means fortranslating the target language hypothesis into the source language;means for adapting the first recognition means using the translation ofthe target language hypothesis into the source language; means foradapting the second recognition means using the translation of thesource language hypothesis into the target language; means for adaptingthe first translation means and the second translation means using thesource language hypothesis; and means for adapting the first translationmeans and the second translation means using the target languagehypothesis.
 17. A computer readable medium having stored thereoninstructions which, when executed by a processor, cause the processorto: recognize, with a first automatic speech recognition component,spoken language in a source language and to create a source languagehypothesis; translate, with a first machine translation component, thesource language hypothesis into a target language; recognize, with asecond automatic speech recognition component, spoken language in thetarget language that is spoken by a translator; create, with the secondautomatic speech recognition component, a target language hypothesis;translate, with a second machine translation component, the targetlanguage hypothesis into the source language; adapt the first automaticspeech recognition component using the translation of the targetlanguage hypothesis into the source language; adapt the second automaticspeech recognition component using the translation of the sourcelanguage hypothesis into the target language; adapt the first machinetranslation component and the second machine translation component usingthe source language hypothesis; and adapt the first machine translationcomponent and the second machine translation component using the targetlanguage hypothesis.